Papers with computation cost
LV-BERT: Exploiting Layer Variety for BERT (2021.findings-acl)
Copied to clipboard
| Challenge: | Modern pre-trained language models are mostly built upon stereotyped development sets . LV-BERT model obtained by our method outperforms BERT on various downstream tasks . |
| Approach: | They propose to exploit layer variety from the layer type set and the layer order to improve pre-trained models. |
| Outcome: | The proposed model outperforms BERT and its variants on various downstream tasks. |
BMInf: An Efficient Toolkit for Big Model Inference and Tuning (2022.acl-demo)
Copied to clipboard
Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie Zhou, Jun Zhang, Jia Chao, Maosong Sun
| Challenge: | Recent years, pre-trained language models (PLMs) have achieved promising results on various NLP tasks. |
| Approach: | They propose an open-source toolkit for big model inference and tuning which can support big model tuning at extremely low computation cost. |
| Outcome: | The proposed toolkit can support big model inference and tuning at extremely low computation cost. |
PipeNet: Question Answering with Semantic Pruning over Knowledge Graphs (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to utilizing explicit knowledge graphs (KGs) are limited by the number of nodes in the subgraph. |
| Approach: | They propose a grounding-pruning-reasoning pipeline to prune noisy nodes in subgraphs to improve the efficiency of graph reasoning with KG. |
| Outcome: | The proposed method reduces computation cost and memory usage while obtaining decent representation of pruned subgraphs. |
A Boundary-aware Neural Model for Nested Named Entity Recognition (D19-1)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition ignore nested entities . a boundary-aware neural model can locate entities precisely by detecting boundaries . |
| Approach: | They propose a boundary-aware neural model for nested named entity recognition which leverages entity boundaries to predict entity categorical labels. |
| Outcome: | The proposed model outperforms state-of-the-art methods on GENIA dataset . it captures dependencies of entity boundaries and categorical labels, which helps to improve identifying entities. |
Incremental Event Detection via Knowledge Consolidation Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to event detection require a fixed set of pre-defined event types . existing methods cannot handle semantic ambiguity and training data imbalance problems . |
| Approach: | They propose a Knowledge Consolidation Network to address these issues . they propose to use a prototype enhanced retrospection and hierarchical distillation to mitigate the adverse effects of semantic ambiguity and class imbalance. |
| Outcome: | The proposed method outperforms the state-of-the-art model by 19% and 13.4% of whole F1 score on ACE and TAC benchmarks. |
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. |
| Approach: | They propose a method that routes queries to RAG or LC based on model self-reflection. |
| Outcome: | The proposed method significantly reduces the computation cost while maintaining a comparable performance to RAG. |
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge (2023.findings-acl)
Copied to clipboard
Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, Bing Xiang
| Challenge: | Open-ended Visual Question Answering (VQA) requires models to reason over visual and natural language inputs using world knowledge. |
| Approach: | They propose a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. |
| Outcome: | The proposed pipeline expands the knowledge coverage from in-domain training data by 4.1% on OK-VQA, without additional computation cost. |
HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual token pruning lack insight into the intrinsic property of the vision encoder . et al., 2017: 99.3% of task accuracy with only 1/3 of the tokens. |
| Approach: | They propose a model-agnostic token pruning method that trains without training . they propose 'HiPrune' method which prunes visual tokens according to their attention . |
| Outcome: | The proposed method achieves 99.3% of task accuracy with only 1/3 of the tokens . it reduces inference FLOPs by 58.7% and maintains 99.99% accuracy with 2/9 tokens. |
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |
SideControl: Controlled Open-domain Dialogue Generation via Additive Side Networks (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate pre-trained language models with attributes are expensive and overfitted on small training sets. |
| Approach: | They propose a novel approach to control the generation of Transformer-based pre-trained language models using a new control attributes loss framework. |
| Outcome: | The proposed method is shown to perform well with very limited training samples. |
Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations (2023.acl-long)
Copied to clipboard
| Challenge: | Using hidden representations, pretrained language models are prone to overfitting due to the huge amount of parameters. |
| Approach: | They propose a method that inserts random autoencoders between hidden layers of a PLM to transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. |
| Outcome: | The proposed method improves performance across sequence- and token-level lowresource tasks. |
Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Transformer-based pre-trained models achieve state-of-the-art results, but they can be prohibitively costly. |
| Approach: | They propose a fine- and coarse-granularity hybrid self-attention that shortens the computational sequence length in self- attention by progressively shortening the computational time. |
| Outcome: | The proposed model reduces computation cost by shortening the computational sequence length in self-attention. |
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2022.acl-long)
Copied to clipboard
Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, Michael Zeng
| Challenge: | Open-Domain Question Answering (ODQA) models typically include a retrieving module and a reading module. |
| Approach: | They propose a new open-domain question-answering framework that uses a knowledge-enhanced version of FiD to improve the approach. |
| Outcome: | The proposed model improves on ODQA benchmark datasets with less than 40% computation cost. |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
PEER: Pre-training ELECTRA Extended by Ranking (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for pre-training require expensive pre-trainer computation cost . ELECTRA model can perform replaced token detection (RTD) task with reduced pre- training cost compared to current models . |
| Approach: | They propose to extend a discriminator-based replaced token detection task into a ranker-based task . they propose to use a binary classifier to perform a more precise task with negligible additional computation cost. |
| Outcome: | The proposed model outperforms state-of-the-art models with ELECTRA in GLUE tasks given the same cost. |
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022.acl-long)
Copied to clipboard
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei
| Challenge: | ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive . |
| Approach: | They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora . |
| Outcome: | The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost. |
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024.findings-acl)
Copied to clipboard
Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, Ge Li
| Challenge: | Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. |
| Approach: | They propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter . they propose a low-rank modulation module to refine per-image features from frozen CLIP backbone . |
| Outcome: | Experiments on four TVR datasets show that the proposed method performs better than other methods. |
Train No Evil: Selective Masking for Task-Guided Pre-Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models can't capture domain-specific and task-specific patterns because of the task-agnostic pre-training stage. |
| Approach: | They propose a task-guided pre-training stage with selective masking between general pre-train and fine-tuning to learn domain-specific patterns. |
| Outcome: | The proposed method can achieve comparable or even better performance with less than 50% of computation cost. |
Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve QA efficiency do not take specific answers into account. |
| Approach: | They propose a transformer-based approach to improve QA efficiency by filtering out questions that will not be answered by the system. |
| Outcome: | The proposed model can approximate the Precision/Recall curves of the target QA system. |
Asymmetric Mutual Learning for Multi-source Unsupervised Sentiment Adaptation with Dynamic Feature Network (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work on pre-trained language models (PrLMs) on labeled sentiment datasets has shown significant improvements on widerange of NLP tasks, including sentiment classification. |
| Approach: | They propose a multi-source unsupervised sentiment adaptation problem with pre-trained features to exploit the extracted pre-train features for efficient domain adaptation. |
| Outcome: | The proposed model outperforms the state-of-the-art methods on multiple sentiment benchmarks and extensive ablation studies to verify the effectiveness of each module. |
CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval (2023.acl-long)
Copied to clipboard
Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen
| Challenge: | Existing multi-vector retrieval methods are slower and require more space to store indices compared to their single-vektor counterparts. |
| Approach: | They propose a multi-vector retrieval method that uses dynamic lexical routing to route different token vectors to the predicted lexicals. |
| Outcome: | The proposed method achieves state-of-the-art performance on several benchmark datasets while being nearly 40 times faster than the current state-out-of the-art method. |
Curating Datasets for Better Performance with Example Training Dynamics (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve data quality but rely on data quantity to improve performance are not effective. |
| Approach: | They propose a method for weighing the relative importance of examples in a dataset based on their Example Training dynamics (ETD) they propose an active learning approach for computing ETD during training rather than as a preprocessing step. |
| Outcome: | The proposed method can be used to improve performance in in-distribution and out-of-distortion testing. |
SConE: Simplified Cone Embeddings with Symbolic Operators for Complex Logical Queries (2023.findings-acl)
Copied to clipboard
| Challenge: | Current geometric-based methods depend on the neural approach to model FOL operators . empirical evidence for explainability is challenging . |
| Approach: | They propose to model conjunction operators using a symbolic modeling approach . they propose to emphasize the essential role of relation projection operator . |
| Outcome: | The proposed method improves answering complex logical queries over previous models. |
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models such as RoBERTa, ELECTRA, and T5 have a large number of parameters which makes them slow and computationally expensive. |
| Approach: | They propose a technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions. |
| Outcome: | The proposed technique saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%. |
Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on monolingual prompts, but we study multilingual prompt for multilingual models. |
| Approach: | They propose a model that uses a unified prompt for all languages, called UniPrompt, to alleviate the effort of designing different prompts for multiple languages. |
| Outcome: | The proposed model outperforms baseline models in the zero-shot cross-lingual setting. |
Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are susceptible to jailbreak attacks, authors say . multimodal information increases the risk of attacks, but also provides additional data . |
| Approach: | They propose a jailbreaking detector that detects maliciously perturbed image inputs . cross-modality information detector is designed to detect cross-modal similarity between harmful queries and adversarial images. |
| Outcome: | a new tool can detect maliciously perturbed image inputs without modification or computation cost. |
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments show that MEO significantly improves computational efficiency . compared to dense networks, sparsely activated networks only employ a few parameters for each input . |
| Approach: | They propose a method that merges multiple experts into one to reduce computation costs . they demonstrate that a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters for each input . |
| Outcome: | The proposed approach reduces the computational cost to that of a single expert by 83.3% compared to 82.6% in vanilla MoE. |
Relation-aware Ensemble Learning for Knowledge Graph Embedding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to explore semantics of knowledge graphs have been proposed to explore these semantics in distinct ways. |
| Approach: | They propose to leverage existing methods in relation-aware manner to learn an ensemble by leveraging existing methods. |
| Outcome: | The proposed method has the same computation cost as general ensemble methods but with much better performance on benchmark datasets. |
Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent research indicates that large language models (LLMs) have demonstrated remark-able capabilities in various programming-related domains, such as code generation and code refinement. |
| Approach: | They propose a framework that combines exploration with refinement to reduce test-time computation overhead. |
| Outcome: | The proposed framework outperforms SOTA and AgentCoder on humanEval and MBPP benchmarks while reducing test-time computation overhead and scalability. |